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Abstract 



It is known that any quantum algorithm for Graph Isomorphism that works within the framework 
of the hidden subgroup problem (HSP) must perform highly entangled measurements across il{n\ogn) 
coset states. One of the only known models for how such a measurement could be carried out efficiently 
is Kuperberg's algorithm for the HSP in the dihedral group, in which quantum states are adaptively com- 
bined and measured according to the decomposition of tensor products into irreducible representations. 
This "quantum sieve" starts with coset states, and works its way down towards representations whose 
probabilities differ depending on, for example, whether the hidden subgroup is trivial or nontrivial. 

In this paper we give strong evidence that no such approach can succeed for Graph Isomorphism. 
Specifically, we consider the natural reduction of Graph Isomorphism to the HSP over the the wreath 
product S„l'Z2- We show, modulo a group-theoretic conjecture regarding the asymptotic characters of 
the symmetric group, that no matter what rule we use to adaptively combine quantum states, there is 
a constant fe > such that no algorithm in this family can solve Graph Isomorphism in e''^ time. In 
particular, such alg orithm s are essentially no better than the best known classical algorithms, whose 
running time is gOCVniogn)^ 

1 Introduction 

Peter Shor's efficient quantum algorithms for order finding, factoring, and the discrete logarithm |Sho94| . 
and Simon's algorithm for determining the symmetry of a type of 2-1 function defined on {0, 1}" |Sim94| . 
led a frenzied charge to uncover the full algorithmic potential of a general purpose quantum computer. 
Creative invocations of the order-finding primitive yielded efficient quantum algorithms for a number of 
other number-theoretic problems HalOS Hal05'. As the field matured, these algorithms were roughly uni- 
fied under the general framework of the hidden subgroup problem, where one must determine a subgroup H 
of a group G by querying an oracle / : G — > S known to have the property that /(<?) = f{gh) h £ H. 
Solutions to this general problem are the foundation for almost all known superpolynomial speedups 
offered by quantum algorithms over their classical counterparts (see IAJL06I for an important recent 
exception) . 

The algorithms of Simon and Shor essentially solve the hidden subgroup problem on abelian groups, 
namely and ZJ^ respectively. Since then, non-abeli an hidden subgro up problem s have received a great 
deal of attention (e.g. IHRTSOOl IHSV VOll IfIM+03I IMRSOU IBGv05I IHMR+06I ). The cases where the 
ambient group G is the dihedral group or the symmetric group possess the most compelling algorithmic 
applications: in particular, we can reduce Graph Isomorphism for rigid graphs of size n to the case of the 
hidden subgroup problem over the symmetric group S2n , or more specifically the wreath product Snl'^2, 
where the hidden subgroup is promised to be either zero or of order two. 

The basic approach to these problems, motivated by the community's success with abelian groups, is 
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to use the oracle to prepare "coset states" pH of the form 

c 

where 15), for a subset S C G, denotes the uniform superposition (l/-\/[5^) Iff)- the abehan 
case, one proceeds by computing the quantum Fourier transform of such coset states, measuring the 
resulting states, and appropriately interpreting the results. In the case of the symmetric group, however, 
determining H from a quantum measurement of coset states is far more difficult. In particular, no 
product measurement (that is, a measurement which treats each coset state independently) can efficiently 
determine a hidden subgroup over S„ IMRS05I : in fact, any successful measurement must be entangled 
over n(nlogn) coset states at once lHMR+061 . 

Unfortunately, our species' evolutionary history has given us little experience in thinking about highly- 
entangled quantum measurements. So far, there are only two known families of such measurements which 
we can actually perform efficiently on a quantum computer: 

Implementation of the Pretty Good Measurement. Pioneered by Bacon, Childs, and van Dam, 
this approach seeks to efficiently implement an explicit measurement, the pretty good measurement. 
For some choices of groups and subgroups, this measurement is known to be optimal IBCvDOSl 
IBCvOSI IMROSbl : for ar bitrary groups, it was recently shown to be sufficient when performed on 
n(log|G|) coset states IHKK06I . On a number of groups, such as the affine, Heisenberg, and 
dihedral groups, implementing the pretty good measurement corresponds to solving random cases 
of a combinatorial problem. However, it is not clear how to carry out this program for the symmetric 
groups, or even whether a corresponding combinatorial problem exists. 

Sieves generated by Clebsch-Gordan decomposition. Kuperberg's algorithm for the hidden sub- 
group problem in the dihedral group |Kup03| starts by generating a large number of coset states and 
subjecting each one to weak Fourier sampling, so that it lies inside a known irreducible representa- 
tion. It then proceeds with an adaptive "sieve" process, at each step of which it judiciously selects 
pairs of states and measures them in a basis consistent with the Clebsch-Gordan decomposition of 
their tensor product into irreducible representations. This sieve continues until we obtain a state 
lying in an "informative" representation: namely, one from which information about the hidden 
subgroup can be easily extracted. We can visualize the sieve as a forest, where leaves consist of 
coset states, each internal node measures the tensor product of its parents, and the informative 
representations lie at the roots. 

The sieve approach is especially attractive in cases like Graph Isomorphism, where all we need to know 
is whether the hidden subgroup is trivial or nontrivial. Specifically, suppose that the hidden subgroup 
H is promised to be either the trivial subgroup {1} or a conjugate of a known subgroup -ffo- Assume 
further that there is an irreducible representation a of G with the property that "^ZhGHg '^W ~ 0; that 
is, a "missing harmonic" in the sense of IMR05a| . In this case, if H is nontrivial then the probability of 
observing a under weak Fourier sampling of the coset state pH is zero. More generally, the probability 
of observing a when measuring any state which is H -invariant — that is, one which is fixed under right 
multiplication by any element of H — is zero. As we discuss below, the sieve process maintains the H- 
invariance of the observed states throughout its operation, so it follows that a cannot appear at any time 
in the algorithm. If, on the other hand, one can guarantee that this sieve process does observe a with 
significant probability when the hidden subgroup is trivial and the corresponding states are completely 
mixed, it gives us an algorithm to distinguish the two cases. 

For example, if we consider the case of the hidden subgroup problem in the dihedral group _D„ where 
H is either trivial or a conjugate of Ho ~ {l,m} where m is an involution, then the sign representation 
TT is a missing harmonic. Applying Kuperberg's sieve, we observe n with significant probability after 
^{s/n) g-j-gpg j£ jg trivial, while we can never observe it if H is of order 2. This framework has been 
successfully applied to groups of the form G" by Alagic et al. |AMR,06] . suggesting that it might apply, 
mutatis mutandis, to a wide class of groups. 

One difficulty in analyzing these sieve algorithms is that the algorithm designer has considerable 
flexibility in how she selects which pairs of representations to combine. We will show below, however, 
that modulo a representation-theoretic conjecture on the symmetric group, the hidden subgroup problem 
related to Graph Isomorphism cannot be solved efficiently by any algorithm in this family. Specifically, 
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no adaptive selection rule produces an efficient sieve whose transcript of observations statistically dis- 
tinguishes the isomorphic and nonisomorphic cases, unless it combines e^^^' coset states. Since the 
best known classical algorithms for Graph Isomorphism run in time gOCViiog") iBabSOlUjLSSI . quantum 
algorithms of this kind offer essentially no improvement over their classical counterparts. 

The paper is organized as follows. We give a brief review of group representation theory in Section|21 
In Section 13 we precisely describe the family of sieve algorithms outlined above. In Sections ^ and |K| 
we describe the probability distributions on the sequence of representations observed throughout a sieve 
algorithm. In Sections |H] and |7] we focus on wreath product groups of the form G1Z2, and show that the 
observed distributions completely fail to distinguish trivial and nontrivial subgroups unless we observe 
a special type of representation. In Section |H| we bound the probability of this event in terms of the 
characters of G. Then, in Section|5] we describe a conjectured bound on the characters of the symmetric 
group, and in Section lTHl we show that this conjecture implies our main result. We conclude in Section lTTl 
by proving that our conjecture holds in a number of cases. 



2 Fourier analysis on finite groups 

In this section we review the representation theory of finite groups. Our treatment is primarily for the 
purposes of setting down notation; we refer the reader to ISer77l for a complete account. 

Let G be a finite group. A representation cr of G is a homomorphism a : G ^ U(V^), where V is a, 
finite-dimensional Hilbert space and U(V^) is the group of unitary operators on V. The dimension of a, 
denoted da, is the dimension of the vector space V. Fixing a representation a : G ^ U(V^), we say that 
a subspace C is invariant if a{g) ■W = W for all g £ G. When a has no invariant subspaces other 
than the trivial subspace {0} and V itself, a is said to be irreducible. 

If two representations a and cr' are the same up to a unitary change of basis, we say that they are 
equivalent. It is a fact that any finite group G has a finite number of distinct irreducible representations 
up to equivalence and, for a group G, we let G denote a set of representations containing exactly one 
from each equivalence class. We often say that each o" G G is the name of an irreducible representation, 
or an irrep for short. 

The irreps of G give rise to the Fourier transform. Specifically, for a function / : G — > C and an 
element cr G G, define the Fourier transform of f at a to be 

/>) = i/^E/(5W5) • 

V sec 

The leading coefficients are chosen to the make the transform unitary, so that it preserves inner products: 
(/i,/2) = E = E . /2(a)) . 

9 o-GG 

If cr is not irreducible, it can be decomposed into a direct sum of irreps n, each of which acts on an 
invariant subspace, and we write a = ti (B ■ ■ ■ (B Tk. In general, a given r can appear multiple times in 
this decomposition, in the sense that a may have an invariant subspace isomorphic to the direct sum of 
Or copies of T. In this case is called the multiplicity of r in the decomposition of cr. 

There is a natural product operation on representations: if A : G ^ U(y) and fj. : G ^ U{W) are 
representations of G, we may define a new representation A(g)/i : G ^ U(F® W) as (A ® /x) (g) : u g) v 1-^ 
A((7)u® /i(g)v. This representation corresponds to the diagonal action of G on V igiW , in which we apply 
the same group element to both parts of the tensor product. In general, the representation A Cg) /i is not 
irreducible, even when both A and fi are. This leads to the Clebsch-Gordan problem, that of decomposing 
A (8 into irreps. 

Given a representation a we define the character of a, denoted Xcr, to be the trace Xcr{g) ~ trcr((;). 
As the trace of a linear operator is invariant under conjugation, characters are constant on the conjugacy 
classes of G. Characters are a powerful tool for reasoning about the decomposition of reducible repre- 
sentations. In particular, when a = 0^ n we have Xtr = X^i X^i and, moreover, for a,T £ G, we have the 
orthogonality conditions 

{Xt,Xt)g = E X<T(5)Xr(5)* = 

' ' 9GG 



1 (J = T 

o- / r 
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Therefore, given a representation a and an irrep r, the multiplicity a-r with which r appears in the 
decomposition of a is (xr, Xa) q- For example, since X>.®ti{9) = ■ Xyj(p)) the multiplicity of t in the 

Clebsch-Gordan decomposition of A igi is (Xt,XaXm)g- 

A representation a is said to be isotypic if the irreducible factors appearing in the decomposition 
are all isomorphic, which is to say that there is a single nonzero in the decomposition above. Any 
representation a may be uniquely decomposed into maximal isotypic subspaces, one for each irrep r of 
G; these subspaces are precisely those spanned by all copies of r in a. In fact, for each r this subspace 
is the image of an explicit projection operator Ilr which can be written 



1^' .ea 

A useful fact is that IIt commutes with the group action; that is, for any h £ G we have 

' ' see sec gee 

Our algorithms will perform measurements which project into these maximal isotypic subspaces and 
observe the resulting irrep name r. For the particular case of coset states, this measurement is called 
weak Fourier sampling in the literature; however, since we are interested in a more general process which 
in fact performs a kind of strong multiregister sampling on the original coset states, we will use the term 
isotypic sampling instead. 

Finally, we discuss the structure of a specific representation, the (right) regular representation reg, 
which plays an important role in the analysis below, reg is given by the permutation action of G on 
itself. Specifically, let C[G] be the group algebra of G; this is the |G| -dimensional vector space of formal 
sums 

s 

(Note that C[G] is precisely the Hilbert space of a single register containing a superposition of group 
elements.) Then reg is the representation reg : G — > U(C[G]) given by linearly extending right multipli- 
cation, reg((;) : h hg. It is not hard to see that its character Xreg is given by 



|G| ,g = 1 , 

3/1 , 



in which case we have (Xreg, X<y) a ~ '^'^ each a £ G. Thus reg contains da copies of each irrep a £ G, 
and counting dimensions on each side of this decomposition implies 

IGl = J2'^' ■ (1) 

ctSG 

This equation suggests a natural probability distribution on G, the Plancherel distribution, which assigns 
to each irrep a the probability Planch ("") = d^/\G\- This is simply the dimensionwise fraction of C[G] 
consisting of copies of a; indeed, if we perform isotypic sampling on the completely mixed state on C[G], 
or equivalently the coset state where the hidden subgroup is trivial, we observe exactly this distribution. 

In general, we can consider subspaces of C[G] that are invariant under left multiplication, right 
multiplication, or both; these subspaces are called left-, right-, or bi-invanant respectively. For each 
a £ G, the maximal a-isotypic subspace is a d^-dimensional bi-invariant subspace; it can be broken 
up further into do- dcr-dimensional left-invariant subspaces, or (transversely) d^ do-dimensional right- 
invariant subspaces. However, this decomposition is not unique. If a acts on a vector space V , then 
choosing an orthonormal basis for V allows us to view cr(p) as a do- x do matrix. Then a acts on the do- 
dimensional space of such matrices by left or right multiplication, and the columns and rows correspond 
to left- and right-invariant spaces respectively. 
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3 Clebsch-Gordan sieves 



Consider the hidden subgroup problem over a group G with the added promise that the hidden subgroup 
H is either the trivial subgroup, or a conjugate of some fixed nontrivial subgroup Hq. We shall consider 
sieve algorithms for this problem that proceed as follows: 

1. The oracle is used to generate I — £{n) coset states pH, each of which is subjected to weak Fourier 
sampling. This results in a set of states pi, where p, is a mixed state known to lie in the maximal 
(Ti-isotypic subspace of C[G] for some irrep ai. 

2. The following combine- and-measure procedure is then repeated as many times as we like. Two 
states pi and pj in the set are selected according to an arbitrary adaptive rule that may depend 
on the entire history of the computation (in existing algorithms of this type, this selection in fact 
depends only on the irreps Oi and Oj in which they lie). We then perform isotypic sampling on 
their tensor product pi (g) pf. that is, we apply a measurement operator which observes an irrep a 
in the Clebsch-Gordan decomposition of cji ® Oj (see |Kup03| or IMROSal for how this measurement 
can actually be carried out by applying the diagonal action). This measurement destroys pi and 
pj, and results in a new mixed state p which lies in the maximal cr-isotypic subspace; we add this 
new state to the set. 

3. Finally, depending on the sequence of observations resulting from the measurements above, the 
algorithm guesses the hidden subgroup. 

We set down some notation to discuss the result of applying such an algorithm. Fixing a group G 
and a subgroup H, let ^ be a sieve algorithm which initially generates I coset states. As a bookkeeping 
tool, we will describe intermediate states of A's progress as a forest of labeled binary trees. Throughout, 
we will maintain the invariant that the roots of the trees in this forest correspond to the current set of 
states available to the algorithm. 

Initially, the state of the algorithm consists of a forest consisting of I single-node trees, each of which 
is labeled with the irrep name ai that resulted from performing weak Fourier sampling on a coset state, 
and is associated with the resulting state pi. Then, each combine-and-measure step selects two root 
nodes, ri and T2, and applies isotypic sampling to the tensor product of their states. We associate the 
resulting state p with a new root node r, and place the nodes ri and T2 below it as its children. We label 
this new node^ with the irrep name a observed in this measurement. 

Thus, every node of the forest corresponds to a state that existed at some point during the algorithm, 
and each node i is labeled with the name of the irrep Oi observed in the isotypic measurement performed 
when that node was created. We call the resulting labeled forest the transcript of the algorithm: note 
that this transcript contains all the information the algorithm may use to determine the hidden subgroup. 

We make several observations about algorithms of this type. First, it is easy to see that nothing is 
gained by combining t > 2 states at a time; we can simulate this with an algorithm which builds a binary 
tree with t leaves, and which ignores the results of all its measurements except the one at the root. 

Second, the algorithm maintains the following kind of symmetry under the action of the subgroup H . 
Suppose we have a representation a acting on a Hilbert space V . Given a subgroup H, we say that a 
state %p £V is H -invariant if o-{h) ■ tp — ij) for all h £ H . Similarly, given a mixed state p, we say that p 
is _ff-invariant if cr{h) ■ p- o-{hy = p or, equivalently, if o-{h) and p commute. For instance, the coset state 
pH is -ff-invariant under the right regular representation, since right-multiplying by any h £ H preserves 
each left coset cH. Now, suppose that pi and p2 are _ff-invariant; clearly pi ® p2 is H-invariant under the 
diagonal action, and performing isotypic sampling preserves _ff-invariance since Ilr commutes with the 
action of any group element. Thus the states produced by the algorithm are _ff-invariant throughout. 

Third, it is important to note that while at each stage we observe only an irrep name, rather than 
a basis vector inside that representation, by iterating this process the sieve algorithm actually performs 
a kind of strong multiregister Fourier sampling on the original set of coset states. For instance, in the 
dihedral group, suppose that performing weak Fourier sampling on two coset states results in the two- 
dimensional irreps aj and ak, and that we then observe the irrep CTj+k under isotypic sampling of their 
tensor product. We now know that the original coset states were in fact confined to a particular subspace, 
spanned by two entangled pairs of basis vectors. 

New nodes is good nodes. 
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Finally, we note that the states produced by a sieve algorithm are quite different from coset states. 
In particular, they belong not to a maximal isotypic subspace of C[G], but to a (typically much higher- 
dimensional) non-maximal isotypic subspace of CfG]®^, where £ is the number of coset states feeding into 
that state (i.e., the number of leaves of the corresponding tree). Moreover, they have more symmetry 
than coset states, since each isotypic measurement implies a symmetry with respect to the diagonal 
action on the set of leaves descended from the corresponding internal node. In the next sections we will 
show how these states can be written in terms of projection operators applied to this high-dimensional 
space. 

4 Observed distributions for fixed topologies 

In general, the probability distributions arising from the combine-and-measure steps of a sieve algorithm 
depend on both the hidden subgroup and the entire history of previous measurements and observations 
(that is, the labeled forest, or transcript, describing the algorithm's history thus far). In this section and 
the next, we focus on the probability distribution induced by a fixed forest topology and subgroup H. 
We can think of this either as the probability distribution conditioned on the forest topology, or as the 
distribution of transcripts produced by some non-adaptive sieve algorithm, which chooses which states 
it will combine and measure ahead of time. We will show, modulo our conjecture, that for all forest 
topologies of sufficiently small size, the induced distributions on irrep labels fail to distinguish trivial and 
nontrivial subgroups. Then, in Section [101 we will complete the argument for adaptive algorithms. 

Clearly, in this non-adaptive case the distributions of irrep labels associated with different trees in 
the forest are independent. Therefore, we can focus on the distribution of labels for a specific tree. At 
the leaves, the labels are independent and identically distributed according to the distribution resulting 
from weak Fourier sampling a coset state itIRTSOO'. . namely 

-^rkf^ <.(/.)) . 

II \heH I 

However, as we move inside the tree and condition on the irrep labels observed previously, the resulting 
distributions are quite different from this initial one. To calculate the resulting joint probability distribu- 
tion, we need to define projection operators acting on CfG]®* corresponding to the isotypic measurement 
at each node. 

First, note that the coset state pu can be written in the following convenient form; 

lie II h6H 

where reg is the right regular representation defined in Section Q that is, pu is proportional to the 
projection operator which right-multiplies by a random element of H , 

" M ^ reg(/i) . 

I I hS-H 

If H is trivial, pH is the completely mixed state p{i} = (1/|G|)1. On the other hand, \i H — {1, m} for 
an involution m, we have 

PH = |^(l-hreg(m)) 
which is proportional to the projection operator 

Hh = i(l + reg(m)) . 

Now consider the tensor product of I "registers," each containing a coset state. Given a linear operator 
M on C[G] and a subset / C [^] = {1, . . . , £}, let denote the operator on C[G*] = C[G]®^ which applies 
M to the registers in / and leaves the other registers unchanged. Then the mixed state consisting of £ 
independent coset states can be written in a number of ways, 

Z^^' = 11(1 + reg(m)<^>) = ^ E 

I I j=i I I /C[f] 
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and this state is proportional to a projection operator on C[G^], 

- ^ + reg(m)*^>) = 1 ^ reg(m)^ . (2) 

3 = 1 IQ\t] 

Note the sum over subsets of regist ers, a them e which has appeared repeatedly in discussions of multi- 
register Fourier sampling |BCvD05l IhMR + 06I |Kup03| IMR,05a1 IMR05cl |Reg02| . 

Now consider a tree T with I leaves corresponding to the I initial registers, and k nodes including 
the leaves. We can represent this tree as a set system, in which each node i is associated with the subset 
li C [I] of leaves descended from it. In particular, /,oot = [(\ and Ij = {j} for each leaf j. 

Performing isotypic sampling at a node i corresponds to applying the diagonal action to its children 
(or in terms of the algorithm, its parents) and inductively to the registers in J^: that is, we multiply 
each register in li by the same element g and leave the others fixed. If at is the irrep label observed 
at that node, let us denote its character and dimension by Xi S'ld di respectively, rather than the more 
cumbersome x^i and dc- . Then the projection operator corresponding to this observation is 

= TTm reg{gY' (3) 

Now consider a transcript of the sieve process which results in observing a set of irrep labels a = {ui} 
on the internal nodes of the tree. The projection operator associated with this outcome is 

n-M = nnr. (4) 

i = l 

We will abbreviate this as 11"^ whenever the context is clear. Note that the various TlJ in the product ijlj 
pairwise commute, since for any two nodes i, j either li and Ij axe disjoint, or one is contained in the other. 
In the former case a^' and b'^ for all a, b. In the latter case, say if U C Ij , we have a'b'^ = b'^ib^^abY', 
and since Xi{b~^cib) = Xi(ffl) it follows from that IlfllJ = Tljllf . 

Given a tree T with k nodes, we write P:^^'[(t] for the probability that we observe the set of irrep 
labels a = {ai} in the case where the hidden subgroup is trivial. Since the tensor product of coset states 
is then the completely mixed state in C[G^], this is simply the dimensionwise fraction of CfG^] consisting 
of the image of 11"^ , or 

Moreover, since measuring a completely mixed state results in the completely mixed state in the observed 
subspace, each state produced by the algorithm is completely mixed in the image of 11"^. In particular, 
if the irrep label at the root of a tree is a, the corresponding state consists of a classical mixture across 
some number of copies of a, in each of which it is completely mixed. Thus, when combining two parent 
states with irrep labels A and ^, we observe each irrep r with probability equal to the dimensionwise 
fraction of A (g) ^ consisting of copies of r, namely 



J'.'®^ = ^(x.,XaX.)g (5) 



(recall that {Xt,X\Xij.) g multiplicity of r in A®/i). We will refer to this as the natural distribution 

in A (g) /X. 

Now let us consider the case where the hidden subgroup is nontrivial. Since the mixed state pf^e can 
be thought of as a pure state chosen randomly from the image of Tiff , the probability of observing a set 
of irrep labels cr in this case is 



where we use the fact that trll^* = [G : H]^ = (|Gj/2)*. Below we abbreviate these distributions as 
Py^^ and P-r whenever the context is clear. Our goal is to show that, until the tree T is deep enough, 
these two distributions are extremely close, so that the algorithm fails to distinguish subgroups of the 
form {1, m} from the trivial subgroup. 
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5 Trivial and legal assignments 



In this section we derive explicit expressions for P^^^ and . First, we fix some additional notation. 
Given an assignment of group elements {a;} to the nodes, for each leaf j we let Ili-^j '^i denote the 
product of the elements along the path from the root to j: 

where the product is taken in order from to the root to the leaf. Then using ||3J and Q we can write 

We say that an assignment {si} is trivial if Ili^j ~ 1 for every leaf j. Then, since trreg{g) = 
Xreg(,g) = |Gj if (/ = 1 and otherwise, we have 

k 

^^'^ = T^*""^ = ^ S ri'^^X.CaO* • (7) 

' ' {ai} trivial »=! 

To get a sense of how this expression scales, note that the particular trivial assignment where fli = 1 for 
all i contributes HiLi <ii/\G\ = Yli 'Ppianch(crOi ^ if the ai were independent and Plancherel-distributed. 
Now let us consider P^. Combining (|5J with © gives the following expression for n-^II^*: 

We say that an assignment {ai} is legal if Ili-^j '^i ^ {^i ^r every leaf j. Then the trace of the term 
corresponding to {ai} is |G|* if {ai} is legal, and is otherwise, and analogous to JTJ we have 

p,^ = ^trn-nr = ^ E n^^x.^r . (9) 



{ai} legal i=l 



Thus these two distributions differ exactly by the terms corresponding to assignments which are legal 
but nontrivial. Our main result will depend on the fact that for most a these terms are identically zero, 
in which case P^ and P^^^ coincide. 



6 The wreath product and its representations 

For any group G, the wreath product G IZ2 is the semidirect product (G x G) xi Z2, where we extend 
G X G by an involution which exchanges the two copies of G. Thus the elements {{a, j3), 0) form a normal 
subgroup K = G X G oi index 2, and the elements ((a, /3), 1) form its nontrivial coset. We will call these 
elements "non-flips" and "flips," respectively. 

The Graph Isomorphism problem reduces to the hidden subgroup problem on Sn 1 1ti in the following 
natural way. We consider the disjoint union of the two graphs, and consider permutations of their 2n 
vertices. Then Sn \ Z2 is the subgroup of 5271 which either maps each graph onto itself (the non-flips) or 
exchanges the two graphs (the flips). We assume for simplicity that the graphs are rigid. Then if they 
are nonisomorphic, the hidden subgroup is trivial; if they are isomorphic, H = {l,m} where m is a flip 
of the form ((a,a~^),l), where a is the permutation describing the isomorphism between them. Note 
that the set of such flips constitutes a conjugacy class. 

For any group G, the irreps of G i Z2 can be written in a simple way in terms of the irreps of G. 
It is useful to construct them by inducing upward from the irreps of ~ G x G (see |Ser77j for the 
definition of an induced representation). First, each irrep of K is the tensor product A (8 p of two irreps 
of G. Inducing this irrep from up to G gives a representation 

(^fA.^} = Ind|(A ® ^) 
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of dimension 2d\dfj,. If A ^ then this is irreducible, and cr{;^^} = o"{p,A} (hence the notation). We call 
these irreps inhomogeneous. Their characters are given by 

ff m ,N fxA(a)xM(/3) +XM(a)XA(/?) if t = 
X{x.^.}[{a,l3),t) = < ift^l ■ ^ ' 

In particular, the character of an inhomogeneous irrep is zero at any flip. 

On the other hand, if A = /x, then o"{a,a} decomposes into two irreps of dimension d\, which we denote 
°^{A A} ^^'^ '-'{X A}- '^^^^ these irreps homogeneous. Their characters are given by 

± ff m ,N /xA(a)xA(/3) ift = 
^ ' [±Xa(q/3) if t = 1 

In the next section, we will show that sieve algorithms obtain precisely zero information that distinguishes 
hidden subgroups of the form {1, m} from the trivial subgroup until it observes at least one homogeneous 
representation. 



7 The importance of being homogeneous 

Suppose that the irrep labels cr = {ai} observed during a run of the sieve algorithm consist entirely of 
inhomogeneous irreps of G J Z2. Since the irreps have zero character at any flip, the only trivial or legal 
assignments {a;} that contribute to the sums Q and @ are those where each ai is a non-flip, i.e., is 
contained in the subgroup K ^ G x G. But the product of any string of such elements is also contained 
in K, so if this product is in = {1, m} where m ^ if, it is equal to 1. Thus any legal assignment of 
this kind is trivial, the sums O and (|5J coincide, and the probability of observing cr is the same in the 
trivial and nontrivial cases. That is, so long as every ai in cr is inhomogeneous, 

P^M^PWM . (12) 

Our strategy will be to show that observing even a single homogeneous irrep is unlikely, unless the tree 
generated by the sieve algorithm is quite large. Moreover, because the two distributions coincide unless 
this occurs, it sufBces to show that this is unlikely in the case where H is trivial. 

Now, it is easy to see that the probability of observing a given representation in G IZ2, under either 
the Plancherel distribution or a natural distribution, factorizes neatly into the probabilities that we 
observe the corresponding pair of irreps, in either order, in a pair of similar experiments in G. First, the 
Plancherel measure of an inhomogeneous irrep o-^x^i^y is 

^^inch('^{A,M}) = ^^2|'gP'' " 2-P^^„eh(^)^^anch(M) ■ (13) 

Similarly, the probability that we observe a homogeneous irrep cr^^ is the probability of observing A 
twice under the Plancherel distribution in G, in which case the sign ± is chosen uniformly: 

GZ ± '^A a 2 

"^planch ('^{A, A}) = Jq^ = 7'planch(A) . (14) 

Now consider the natural distribution in the tensor product of two inhomogeneous irreps (T{a,a'} and 
The multiplicity of a given homogeneous irrep cr^^ in this tensor product factorizes as follows, 

X{V,r}.X{A,A'}X{M,M'})^, = T; ({Xr,XAXM)G (Xr,XA'XM')G + (Xt,XaXm')g (Xr,XA'XM)G) ' 

Thus the probability of observing either ct^ or a~^^ under the natural distribution is 

p<'j^A,A't®<'{^,^'} _ j_ /pA^MpA'^fj' _|_ pA®f.'pA'®M\ ^25) 
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In other words, the probabihty of observing a homogeneous irrep of G1X2 is the probabihty of observing 
the same irrep in two natural distributions on G. Let us denote the probabihty that we observe the same 
irrep in the natural distributions in \® n and A' (8) ^j! — that is, that these two distributions collide — as 

pA®/^,A ®fi \ ^ pA®/^ pA'®^t' 

T 

Then (I15I I implies that the total probability of observing a homogeneous irrep is 

^± 2 \ ^^^^ t^oll j — Y coll ' coll j ' V / 

In the next section, we will show that if A, ^l, A' and ji have certain character-theoretic properties, then 
no irrep r occurs too often in any of these natural distributions, and so the probability of a collision is 
small. 



8 Collisions, smoothness, and characters 



Let us bound the probability Pcoii ~ 



co\i^''^ that the natural distributions in A (gi /x and A' ® jJ.' 
collide. The idea is that Pcoii is small as long as both of either or both of these distributions is smooth, 
in the sense that they are spread fairly uniformly across many r. The following lemma shows that this 
notion of smoothness can be related to bounds on the normalized characters of typical representations. 

Lemma 1. Given a group G, say that an irrep \ of G is /(n)-smooth if 



E 

9GG 

Suppose that A and /i are f{n) -smooth. Then 



Xa(5) 



dx 



<iin} 



maxrdr rj]— 
Pcoii < ^=^\/ f[n 



Proof. We have 



|G| 

^ pA»MpA'»M' < inaxP; 



Using and using Cauchy- Schwartz twice gives 



pAQ3/^ 

< 



rfA^M 
dr 

d\d fj, 

dr 
dxd^ 

dr 

d\d^ 

dr 



{Xr ? XaXm) G 

\/ {Xr,XT)a (xaXm,XaXm)g 
\J (IxaI^ , IxmOg 



E 



Xa(5) 



dx 



E 

SGG 



xM 



\G\ 



□ 



Now let us focus on the case relevant to Graph Isomorphism, where G = Sn. The reader can find an 
excellent introduction to the representation theory of 5^ in |JK81| . Here we recall that each irrep of the 
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symmetric group Sn corresponds to a Young diagram, or equivalently an integer partition Ai > A2 > • ■ • 
where = ^- The number of irreps is thus the partition number, 

pin) = {l + o{l))-^e'^ <e'^ (17) 
4v3 • n 

where 

S = vWStt . 

The maximum dimension of any irrep is bounded by the following result of Vershik and Kerov: 
Theorem 2 r| VK85p . There is a constant c > such that for all n> 1, 

T 

In this case, Lemma Ogives 

Pcoii < e-(^/^)^v7R • (18) 
Therefore, our goal is to show that typical irreps of Sn are /(n)-smooth where /(n) grows slowly enough 
with n. Indeed, to prove our main result, even /(n) = for some a < c would suffice. However, a 

well-motivated conjecture, described in the next section, implies /(n)-smoothness with /(n) — 0{^/n). 

9 A conjectured character bound 

In this section we present a conjecture regarding the high-dimensional irreps of S„. First we fix some 
notation. Given a permutation n G Sn, let s{n) denote its support, i.e., the number of objects not fixed 
by it. Let c{n) denote the number of nontrivial cycles of which tt is composed, i.e., other than fixed 
points, and let t{n) denote the length of the shortest string of transpositions whose product is tt. Since 
a single fc-cycle requires fc — 1 transpositions, we have t(7r) = s{tt) — c{n). 

Then the following conjecture states that for all irreps whose dimension is sufficiently close to the 
maximum dimension, their normalized characters obey a certain uniform bound. 

Conjecture 1. Say that an irrep A of Sn big if 

Then there is a constant A such that, for n sufficiently large, the normalized character of all big A obeys 



XA(7r) 



dx 



for all TT £ Sn- 

We present evidence for this conjecture in Section [TTI below. For now, we point out that it has the 
following two important consequences, which will imply our main result. 

Lemma 3. Conjecture^implies that all big irreps A are 0{^/n) -smooth. 

Lemma 4. Say that an irrep A of Sn is really big if 

-(l/2)y5Ilogn /-J 



Suppose that A and /i are really big, and that t is selected according to the natural distribution @ 
A (8) /i. Then Conjecture^implies that r is really big with probability 1 — e~'^'^' . 

We will prove Lemmas and 0] with the help of the following combinatorial lemma. 

Lemma 5. Let < z < 1/n, and let 



qn{z) = 

Then 



= • (19) 
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Proof. Consider the exponential generating function 

g{y,z) =J2 7;;ji--^^ 



Using the techniques of |Wil94l Chapter 3], we can write this as a product over all k of contributions 
from the (fc — 1)! possible fe-cycles, including fixed points. Since each such cycle contributes fc to m and 
A: — 1 to t{n), and since there are (fc — 1)! fc-cycles on a given set of k objects, it follows that 



oo i. i. _ 1 



= exp — ln(l -yz)] = 



Now note that e~"g{n, z) is the expectation of qm{z), where m is Poisson distributed with mean n. Since 
qm{z) > 0, this expectation is at least qn{z) times the probability that m — n, which is e~'^n'^/n\ = 
(1 — o(l))/\/27rn. Thus we have 



q„{z) < (1 + o{l))V2nn ■ e "g{n,z) 



which concludes the proof. 



□ 



Note that since the Poisson distribution is tightly peaked, if qn{z) varies smoothly with n then 
e~"g{n,z) converges to qn{z) as n — > oo. In that case we could strengthen Lemma |S] by removing the 
0(\/n) term; but, since this is not necessary to our result, we have not attempted to prove it. 



Proof of Lemma\^ If the conjecture holds, then we have 



E 



Xa(7I-) 



(20) 



where C = A^. Setting z = C/n^ in Lemma |3 gives 

g„(C/n2) =0(^)e-" 
Using — log(l — x) = X + x^ /2 + x'^/3 + • • • , we have 



1 



C 



e exp 



C 



O 



:0(1) 



which completes the proof. 



□ 



Proof of Lemma^ Using simple dimension-counting arguments as in IMRSOSj . the multiplicity (xr, XaXh)g 
of r in A ® /i is at most dr. First we show that we can assume that r is big; if it is not, by we have 



^2 



- ^/n log n 



dxd^ ^f, - (1/ 2) log ny 

Now, to bound the probability that r fails to be really big, we rewrite © as follows: 



(21) 



^ Xr XaXm 

^ dr dx dfi 



(recall that the characters of Sn are real). Since t is big, Conjecture applies to it as well as to A and 
/i. In that case we can bound P^®'' using the triangle inequality. 



" - n! ^ 



2ClXA Xm 

dr dx df^ 



<^Y1 (.l*(^'n-*W/^)^ ^ £^g4c/n^/2) 
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where C = . Setting z — C /'n?^'^ in Lemma |3 then gives 



Again using — log(l — x) — x + x /2 + a; /3 + • ■ • , we have 

c \-"'/^ _„ // c IC^ 



The probability that r fails to be really big, conditioned on the event that it is big, is thus less than 

^ 0{^ffi) ©(e*^/"*)^) e*^ = e-v^i°g" e°(^' = e-"'^' . (22) 
Combining (|2IJ and ij^U, the probability that r fails to be really big is e~^^^\ □ 

10 Proof of the main result 

We are now in a position to present our main result. 

Theorem 6. Let c be the constant of Theorem\^ Then Con-iectureUl implies that for any constants b,c 
such that b < c/2 and c < (c/2) — 6, no sieve algorithm which combines less than coset states can 

solve Graph Isomorphism with success probability greater than e~'^^ . 

Proof. Let us first consider the behavior of a sieve algorithm A in the case where the hidden subgroup 
H d Sn 1^2 IS. trivial. For convenience, extend the notion of "really big" to representations of Sn 1 1^2 
by declaring ""{a.^} ^'^ t>e really big when both A and p are. We will establish that with overwhelming 
probability, all the irrep labels observed by A axe both really big and inhomogeneous. 

Let i < e^^ be the number of coset states initially generated by the algorithm. We begin by showing 
that with high probability, the irrep labels on the £ leaves, i.e., those resulting from weak Fourier sampling, 
are all really big and homogeneous. If H is trivial, then these irrep labels are Plancherel-distributed, 
and by 11311 the probability that a given one fails to be really big is at most twice the probability that a 
Plancherel-distributed irrep of Sn fails to be. The Plancherel measure of the set of non-really-big irreps 
of Sn is at most the number of irreps times the measure of a single such A, so by 1171 this probability is 
at most ^ 

2p(n)% < 2e^^e~^'°'^" = (^) . 
n'. 

Moreover, by 1141 the probability that the label of a given leaf is homogeneous is the probability that 
we observe the same irrep of Sn in two samples of the Plancherel distribution. Using Theorem |21 this is 



y ^ < max ^ < e 

\ Til / i -nl 



Thus the combined probability that any of the £ leaves have a label which is not both really big and 
inhomogeneous is 



Now, assume inductively that all the irreps observed by the algorithm before the ith combine-and- 
measure step are really big and inhomogeneous, and that the ith step combines states with two such 
labels o"{A,A'} and o"{^,^'}. Recall from 1161 that the probability this results in a homogeneous irrep 
is bounded by the probability Pcoii of a collision between a pair of natural distributions in Sn- Then 
Conjecture Q Lemmas and |3] and Theorem |21 imply that this probability is bounded by 

Pcoii<e-(^/^'^0(n^/^) . 
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In addition, Conjecture and Lemma 2] imply that the the probability the observed irrep fails to be 
really big is e-'^^^. Since each combine-and-measure step reduces the number of states by one, there 
are less than I such steps; taking a union bound over all of them, the probability that any of the observed 
irreps fail to be both homogeneous and really big is 

I (e-(^/2)^0(n^/^) + e'"(^') < e(^-(=/2)+o(i))y5T (34) 

Let us call a transcript inhomogeneous if all of its irrep labels are. As c < (c/2) — 6 by assumption, 
combining 1231 and 1241 1 . for n sufficiently large, yl's transcript is inhomogeneous with probability at 
least 1 - e"'^^. 

Now consider yl's behavior in the case of a nontrivial hidden subgroup, H = {l,m}. Inductively ap- 
plying Equation 1121 shows that the probability of observing any inhomogeneous transcript is exactly the 
same as it would have been if H were trivial. Thus the total variation distance between the distribution 
of transcripts generated by A in these two cases is no more than e"'^^, and the theorem is proved. □ 



11 Evidence for the conjecture 

In this section we give some evidence for our conjecture, by showing that it holds in a variety of cases. 
First we prove the following lemma, which bounds the height and width of a Young diagram corresponding 
to a big representation. 

Lemma 7. //A is big, then the width and height of its Young diagram are less than l-^nlogn. 

Proof. The Robinson-Schensted correspondence |Ful97| maps permutations to Young diagrams in such a 
way that the uniform measure on Sn maps to the Plancherel measure. In addition, the width (resp. height) 
of the Young diagram is equal to the length of the longest increasing (resp. decreasing) subsequence. 

It follows that if A has width w, its Plancherel measure d\/n\ is at most the probability that a random 
permutation in Sn has an increasing subsequence of length w. By Markov's inequality, this probability 
is at most the expected number of such subsequences, which is 

w I w] \ w 

since w\ > w^e~^ . If w — 4y'nlog7i, comparing this to the Plancherel measure of a big irrep A gives 

,2 / p \ S^logn 

g-2v'nlogn ^ < I 1 ^ ^4 /g-\-8\/"logn 

n! ~ \ 4 log n y 

and since e < (4/e)^ this is a contradiction. The argument for the height is identical. □ 

Our next lemma uses deep results of Kerov |Ker03) and Biane |Bia98j to show that our conjecture 
holds for permutations whose support grows slowly enough with n. Indeed, it was Biane's work, which 
builds on recent results in the theory of noncommuting random variables, that originally inspired our 
conjecture. 

Lemma 8. Conjecture^holds for all permutations whose support is less than (log n)^~'' for some f3 > 0. 

Proof. We first use a beautiful fact due to Kerov IKerOSI . Any sequence of irreps A„ of S„ whose 
dimensions differ less than exponentially from the maximum dimension as n — > 00, that is, such that 
d\^ = e~°''"'\/nT, have Young diagrams which converge to an asymptotic shape when scaled by ^n. This 
condition is clearly satisfied by any sequence of big irreps; for aesthetic reasons, we show this asymptotic 
shape in Figure 

Note that these Young diagrams have width and height thus solving Ulam's celebrated problem 

of the likely length of the longest increasing or decreasing subsequence in a random permutation. However, 
the notion of convergence used in IKer03| does not rule out the possibility that the Young diagrams 
corresponding to big A have long thin "tails" near the horizontal and vertical axes, so we rely on Lemma|7| 
to bound the length of these tails. 
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Figure 1: The asymptotic shape of a Young diagram associated with an irrep A of Sn whose dimension is 
close to the maximum dimension. It is scaled by ^/n, so its height and width are 

Now, Theorem 1.3 of IBia98l and the surrounding discussion establishes that for any sequence of 
irreps A whose Young diagrams converge to some asymptotic shape, and which moreover have height and 
width o{'n}^'^^'^) for all e > (which follows in our case from Lemma[7|l the normalized character of any 
permutation with small support can be approximated by a product over its cycles. Specifically, for any 
asymptotic shape there is a set of constants at for fc > 1, called the free cumulants associated with that 
shape, ^ such that, if s{-k) < (logn)^"'' for some fixed /3 > 0, 

XA(7r) -t(ir)/2 TT c,-(ir) , , ~t(Tr)/2s 

— -j—^ — n ^ " "-k + ''('^ ) 

^ fc>l 

where Cfe(7r) is the number of cycles of size k in vr. For the Kerov shape in particular, ai — 1 and = 
for all k > 2, giving a bound even tighter than our conjecture, 

for any tt other than the identity. □ 

The next two lemmas use simple counting arguments to show that Conjecture^holds for permutations 
of large enough support, or which contain large cycles. 

Lemma 9. Conjecture^holds for all n with tiji) > ^/n\ogn. 

Proof. We use the Murnaghan-Nakayama formula for the character |JK81I . A ribbon tile of length k is 
a polyomino of k cells, arranged in a path where each step is up or to the right. Given a Young diagram 
A and a permutation tt with cycle structure fci > A;2 > • • • , a consistent tiling consists of removing a 
ribbon tile of length fci from the boundary of A, then one of length k2, and so on, with the requirement 
that the remaining part of A is a Young diagram at each step. Let hi denote the height of the ribbon 
tile corresponding to the ith cycle: then the Murnaghan-Nakayama formula states that 

XAW=En(-l)''^' (25) 

T i 

where the sum is over all consistent tilings T. 

Clearly the number of consistent tilings is an upper bound on |xa('7'")|. Now, we claim that for any 
fixed k, the number of possible locations for a ribbon tile of length k on the boundary of a Young diagram 
A of size n is less than \/2n. To see this, associate each one with the cell of A which is directly above 

^Readers should be warned that our notation difEcrs somewhat from Bianc's. 
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Figure 2: We associate each possible location for a ribbon tile of fixed length k with a cell (shaded) which 
is above the tile's lower end and to the left of its upper end. The resulting sequence of cells moves up and 
to the right at each step, implying that the number of locations is less than \/2n. Here k = 3. 



the tile's lower end, and directly to the left of its upper end, as shown in Figure |5| A little thought 
reveals that the resulting sequence of cells has the property that each one is above and to the right of 
the previous one. Therefore, if there are i locations, we have 

t 

i=l 

and so ^ < \/2n. It follows that the number of ways to remove the ribbon tiles corresponding to the c(7r) 
nontrivial cycles is less than 

Moreover, after these ribbon tiles are removed, the number of consistent tilings of the remaining Young 
diagram is simply the dimension of the corresponding irrep of S„^s{-7r), which is less than \/|S„_s(^)| — 
\J (n — s(7r))!. Therefore, if A is big we have 

XA(7r) 



^ (2n)=W/V('i-K^))! 



Here we used the bound (n — s)!/n! < 4 • n~ 
c(7r) < t(7r), s(7r) < 2f(7r), and f(7r) = s(7r) - c(7r). Finally, if tiji) > ^/T^logn, the term e^^°^" can be 
absorbed into and Conjecture holds for any A > \f2^ . □ 

The next lemma shows that Conjecture also holds for permutations that contain sufficiently large 
cycles, for the simple reason that in this case the character is zero. 

Lemma 10. Conjecture^ holds for all n that contain a cycle of size 8^/nlogn or greater. 

Proof. The maximum size of any ribbon tile that can be placed on the boundary of a Young diagram is 
its width plus its height (minus one). Thus if such a cycle exists, by Lemma |7] there are no consistent 
tilings and the Murnaghan-Nakayama formula 12511 implies that the character is zero. □ 

Unfortunately, Lemmas |H1 ISl and 1101 leave uncovered the case of permutations whose support is 
between, roughly, logn and y^rilogn and which consist entirely of short cycles. However, it seems very 
plausible to us that the scaling we conjecture here extends across this gap. 

Moreover, it should be noted our main result still holds even if the log n term in our definition of big 
representations were replaced by a more slowly-growing function of n such as log log n. Thus we can get 
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by with significantly weaker forms of the conjecture, which apply only to representations even closer to 
the maximum dimension than we consider here. 
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